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Abstract 

This paper addresses the problem of effident instructioQ sup- 
ply in superscalar architectures. The presence of branch 
instructions introduces two types of overhead on instzuctioD 
supply efficiency, namely the delay required to resolve branch 
decisions and the fetch penalty due to poor code alignment 
The former can be reduced by conventional branch prediction 
techniques. The latter, however, has not been well addressed 
in past research. We propose the instruction trace buffer 
tedinique to aUeviate the alignment problem. An instruction 
trace buffer is an aggressive extension of the conventional 
loop buffer technique. It caches recent instruction traces in 
a circular buffer to predict branch behavior as well as to 
improve code alignment. This s^proach relies on the fact 
that dynamic branch behavior is stable in most cases. Ap- 
plication traces are coUected to verify this assumption and 
to evaluate our scheme. The result indicates that instruction 
trace buffers lead to substantial improvement on instruction 
supply efficiency for numerical applications. For these ap- 
plications near perfea (99%) efficiency is achieved without 
imposing undue demand on the instruction memory band- 
width, which is essential in other hardware based alignment 
techniques. 

Introduction 

RISC processors have t>een successful in achieving an in- 
struction issue rate of nearly one per clodc cyde. To boost 
processor performance still further, superscalar architectures 
seem to be the next logical step. Similar to pipelined proces- 
50IS, data and control dependency are two nu^or obstacles 
to substantial speedup, cfynamic scheduling techniques can 
be used to resolve data dtependency. In these schemes the 
processor attempts to issue instructions continuously with- 
out waiting for the previous ones. TTie effectiveness of this 
approach is still limited by control dependency, since in- 
stmction supply must be h^ted until branch instructions are 
resolved Therefore control dependency is really the major 
limiting factor for superscalar architectures [1] [2][3]. 

Branch prediction techniques are usuaUy used to deal with 
control dependency. In these schemes the processor takes a 
guess whenever a branch instruction is encountered. Study 
has shown that we can actually come up with good guesses by 
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remembering the past [4]. Hardware based branch prediction 
achieves a predicticm rate of up to 90%, dq>ending upon how 
much infoimation is remembered. Wth some analysis and 
profiling, software-based branch prediction achieves roughly 
the same accuracy as dynamic, hardware based schemes [5]. 
However the responsiveness of software-based techniques 
is questionable: given the predicted branch outcome, the 
processor still has to wait for the correct target address. A 
complete software solution to this problem means a drastic 
dtange in the imderlying inscructioh set, which is more or 
less against the spiht of superscalar architectures. 

Good guesses only solve the first part of the problem. Even 
deterministic branches can do barm to fetch efficiency be- 
cause they disturb the locality of instruction accesses. There 
is a penalty for discontiguous instruction fetches because the 
instruction memory is block structured. This is illustrated 
in Figure 1. In this scenario the instructions after the taken 
branch and those before the branch target have to t>e thrown 
away. As will be seen later in this paper, instructions thrown 
away in this fashion constitute a significant portion of the 
branch penalty. 

Delay slot schedtiling and cmipile time code layout may help 
solve the alignment problem. However, study has shown that 
it is very difficult to fill more than one delay slot [5]. For 
architectures with aggressive lookahead c^a^ility as super- 
scalar processors, this is not sufficient. Moreover the num- 
ber of delay slots is variable in our case, depending upon 
the alignment of branch instructions. This poses some code 
compatibility problems. The effectiveness of compile time 
code alignment seems to be limited, because restricting the 
layout of certain instructions may destroy code continuity in 
other cases (a good example is a code sequence containing 
two bran^ targets). A feasible solution is to have the fetdier 
align instructions for the decoders on the fly, which imposes 
extra bandwidth requiren^t for the instruction nmnory - the 
fetcher must fetch more instruction words than necessary, se- 
lect the useful ones, and then merge them for the decoders. 
Here the gain may not justufy the expense [2]. 

The aligtunent problem occurs mainly t>ecause instructions 
can not be aUgnied conveniently within a block-structured in- 
struction memory, ev^ if we could predia perfectly which 
direction they go at run time. Tb efifectively deal with the 
alignment problem we propose the instruction trace buffer 
(TB) organization. The idea of the instruction trace buffer 
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is to cache leceni insmictioD traces ' in a circular buffer. 
Similar to a conventional loop buffer [7] [6]* the TB starts 
operating when a badnp^ard branch into the buffer is de- 
teaed. Subsequent instructions are deUvered by the TB until 
a branch misprediction occurs. Instruction accesses in the 
trace buffer are always sequential legardless of the existence 
of PC-changing instructions. To handle long loc^ contain- 
ing several instruction runs without demanding full associa- 
tivity of the buffer, a small associative cache (called tbe entry 
cache) is used for loop detection. By compacting instruc- 
tions into one sequential trace the TB achieves good code 
alignment; by delivering instructions according to previous 
branch outcomes it performs simple branch prediction. 

The rest of the p^r will be organized as follows. We first 
state the hardware assumptions and the application traces 
used for analysis throughout the paper. We then analyze the 
branch behavior using these traces and show the potential 
benefits of our approach. After giving the motivations we 
move on to the detailed mechanism and implementation of 
instruction trace buffers. A traoe-^lnveD simulator is writ- 
ten to evaluate our design and to compare our scheme with 
conventional instruction memory with branch prediction ca- 
pability. Finally we discuss the simulation results and give 
our conclusions. 

Hardware Assumptions and Application TVaces 

We will use a pipelined RISC processor as the backbone of 
our hypothetical superscalar architecture. The only pipeline 
stages of interest to us are the instruction fetch stage (IF) and 
the instruction decode (DE) stage. The structure of these two 
stages are very sinular to those in the MIPS R2000 processor 
[8] or the Motorola 88100 processor [9]. 

Several Instructions air fetched into the processor in the IF 
stage. Because the way most instruction caches are struc- 
tured, instruction fetches can not cross the instruction block 
^ boimdary. We take the number of decoders as the size 
of the instruction block. By the begiiming of the next cy* 
de these instructions are ready for decode in the ID stage. 
Branch conditions are calculated explidUy with compare in- 
structions, and the results are stored in tiie gerual purpose 
registers for later use. The hypothetiocal processor contains 
dedicated hardware for calculating the branch target address 
and the branch direction. Therefore the branch can be re- 
solved before the end of the D£ stage. The penalty for a 
misprediaed branch is one processor cycle. ^ 

In addition we require that the processor have the ability to 
nullify invalid instructions in the pipeline when a mi^rc- 
dicted branch is detected. This assurc^tion is true for all 
schemes involving speculative instruction fetches. However 
there is no danger of causing inconsistent register state, be- 
cause none of the invalid instructions can execute beyond the 
D£ stage. Detailed timing assumptions are given in Figure 
2. 
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tiont between taken bnnches, whereas an instnictioD tnce is ttie 
iequentially "^executed" codes ended with a backward branch to the 
fftan of the tnce. 
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Figure 2: Instruction Fetch and Decode Tinting 



The implication traces are colleaed on a MIPS machine using 
iht pixie trace facility. Inputs are drawn firom the sample data 
files associated with the benchmaik programs. Programs 
were run only long enough to suppress the stait-up effea 
(this is observed when the statistics starts to stablize). T^le 
1 describes the application traces used in this paper: 

Dynamic Branch Behavior Measoranents 

The ultimate goal of instruction trace buffers is to optimize 
instruction fetches for loops with modest size, which is the 
common case claimed by the well known 90/10 rule. 7b give 
solid proof to the validty of our s^roach we ran a series of 
benchmark programs and obtained the measurements shown 
in Figure 3. We call an loop iteration stable if the instruction 
sequence executed therein is exactiy the same as the one in 
the previous iteration. We call an lo<^ iteration fresh if it is 
not stable. From these results we come up with the foUowing 
observations: 



• Over 99% of the instructions arc delivered by loops 
with sizes smaller than 1024 instruction wortSs (except 
for the dinero benchmark). This means we can achieve 
substantia] gain if we optimize for these loc^. The 
result also shows that we can cq)ture most of the stable 
loops with fewer than 128 instructions. 

♦ In average around 50% of the instructions are delivered 
by stable iterations. That is, recent instruction traces can 
be reused 50% of the time. This gives an estimate of how 
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T^le 1 : Benchmark programs 
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Figure 3: Dynamic Branch Behavior 
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often the TB would be activated at ran time. However 
this ratio varies with different application programs. 

• A large portion of the codes (50% in average) are still 
delivered by fresh iterations. These instructions evade 
the TB optimization and must be handle by conventional 
branch prediction tedmiques. 

• 2-13% of the dynamic instruction count is constributed 
by taken branches. Above 50% of these are from sta- 
ble iterations. This may not seem to be a major plus 
for the TB technique at first glance. However each of 
these taken branches is going to translate into several 
instnictions of alignment penalty when the TB is not 
used. This indicates that the saving achieved by a TB 
can be substantial. 



From the above statistics it is reasonable to conclude that the 
TB is not itself a complete solution. However it can be used 
to boost the perfonnance of conventioal branch prediction 
techniques. In the next section we describe the detailed 
mechanism of instruction trace buffers. 

Mechanism of Instruction IVace Buffers 

The idea of TB is motivated by the fact that most codes 
appear to be more predictable than they appear. For example, 
consider the following C program that performs an in-order 
traveral of a binary tree. The code sequence executed at run 
tune is also shown (read from the leftmost colunm to the 
rightmost column). 



Entry Cache 



** program ** 

int inorder<p) 
btree *p; 



code executed ** 
1 
3 

{ 111 
1: if (p->data~mJLL> 3 2 3 

2: return; 14 1 

3 : inorder (p->left) ; 3 5 3 

4: printf {"%d",p->data) ; . 1 

5: inorder (p->right) ; , 3 

6: return; 

} 

Hie program behaves like a loop with an if-then constructs, 
with one branch of the if-statement taken more frequently 
than the other. If no optimization is done we have to pay 
for the same branch overhead again and again. However, 
if we could somehow cache the "preferred" loop iteration 
(containing statement 1 and 3 in this example), the saving 
can be significant. Now we present the structure of a TB and 
the mechanism required to achieve this goal 

Instruction T^ace Buffer Organization 

The organziation of a TB is presented in Figure 4. It con- 
sists of a circular instruction buffer and an entry cache. The 
instruction buffer holds the trace instructions, the predicted 
branch directions, and the prediaed target addresses. The last 
item is required because the target address of a branch instruc- 
tion may change over time (which is typical for subroutine 
returns). The entry cache stores a list of address/poinierpairs. 
The address part denotes the location of a backward branch, 
and the pointer gives the position of the branch target in the 
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Figure 4: Organization of a Trace Buffer 
The figure assumes an 8-slot entry cache and a o4-entdes 
instuction buffer. The instruction buffer behaves like a FIFO 
buffer during trace collection. During trace emission the 
entry pointer and the write pointer define the boundary of the 
cached loop iteration. 



instniction buffer. Looping is deteaed when the program is 
about to jump to first instruction of the loop iteration cadied 
in the TB. An example of trace storage in a IB is given in 
figures. 

The use of an entry cache is to lift the associativity require- 
ment in conventional loop buffers. Since the size of a TB 
must be large enough to hold instruction traces, we can not 
afford to have a fuUy associative TB. The cost of implement- 
ing such a TB and the time required to access it will make the 
2q)proach infeasible. Therefore we choose to separate loop 
detection from loop storage. This implies that only several 
instructions in the trace can be designated as loop entries. 
Our simulation results indicate that this is not a serious lim- 
itation. In our simulations we use an entry cache containing 
fewer than 8 entries, and we observe no substantial cache 
replacement activities - most of the entries are invalidated 
before they can be replaced due to the limited instruction 
buffer size. For the same reason a simple FIFO replacement 
policy suffices. 

The instruction buffer does not require full addressing gener- 
ality. All read and write accesses to it are sequential. There- 
fore we choose to organize the instruction buffer into several 
banks, with the number of banks equal to the number of de- 
coders in the processor. Since the reads and writes may start 
with any position in the buffer, we need extra logic to route 
the correct data item to/ffom the various banks. Figure 6 
and Figure 7 suggest an implementation of the bank address 
generation logic and the data route logic. 

The speed requirement of a TB is roughly the same as that 
of a register file in RISC processors - each basic operation 
(defined below) must be completed in half an cycle. Although 
the size of the instruction buffer is xisually larger than that 
of a register file, we believe the speed requirement can be 
satisfied by the proposed implementation. 

The basic operations on the TB is summarized below: 

• WRITE: write a sequence of instructions to the buffer, 
starting from the position indicated in the write pointer. 
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Instruction addresses in the trace: 

100 101 110 111 112 120 121 122 123 124 110 



taken branch taken branch 



backward branch 



Instruction buffer content: 

#0 #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 
instruction code; <omitted> 
taken/untaken: 0100100001 
target address: - 110 - - 120 - - - • no 

Write pointer (designating the end of the loop iteration) : 10 
Entry pointer (designating the start of the loop iteration) : 2 
Entry cache content: { (124,2) ) 

Figure 5: Trace storage example. 
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Figure 6: Implementation of a Trace Buffer 
The implementation given in this figure is meant for a 4- 
decoder superecalar processor. The instruction buffer is or- 
ganized into 4 banks, each with its own address generation 
and data route logic. Simultaneous accesses to the instruction 
buffer will always be directed to different banks because of 
their sequentiality. 
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Figure 7: a slice of the address generation logic and the data 
route logic 

The data fed in from the r/w ports are multiplexed to the 
propriaie banks, the offset value designates the port number 
of the first valid instruction, whose buffer address is given 
by the read/write pointer. The selected data item is then read 
from or written to the bank address. 



These instructions and the associated prediction infor- 
mation are supplied by the instruction cache. Note that 
the instruction buffer is maintained in a circular fashion. 

• READ: read a sequence of instructions from the buffer, 
starting from the position indicated in the read pointer. 
If the read has gone past the last instruction of ttie current 
loop iteration, wrap around to the begiiuiing of the loop. 

• ADVWPTR:advancethe write pointer Also invalidate 
stale data in the entry cache 

• ADVRPTR: advance the read pointer. 

• LOOKUP: look up an address in the entry cache. 

• ADD: add an address/pointer pair to the entry cache. 

• ROLLBACK: rollback the write pointer when a mispre- 
diction occurs. 



♦ SETPTRS: adjust the positions of the read and entry 
pointers when the program enters a trace 

• CORRECT: correct the prediction related information in 
the buffer. This happens when a branch is misprediaed. 

To ^sure uninterrupted instruction supply some precaution 
must be taken. Bank collisions may occur when we attempt 
to deliver instructions from different repetitions of the loop 
iteration. This is because the buffer addresses of these in* 
structions are not contiguous. We solve this problem by 
appending the first several instructions to the end of the loop 
iteration. This guarantees that buffer accesses are always 
contiguous and thus no bank conflict can occur. Care must 
be taken when we wtdp around the read pointer so that the 
instructions impended will not be redelivered. 

Integrating the TB and the Superscalar Processor 

In this section we discuss the interaction between the TB and 
other components in a supersalar architecture. The blodc 
diagram of an integrated superscalar architecture is shown 
in Figure S. The processor contains an instruction cache 
(possibly equipped with branch prediction hardware), a set 
of decoders (foiu* in this example), the execution units, and 
the TB. Aside from the instruction registers extra t>uffers are 
required to hold the the branch prediction information and 
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Figure 8: Proposed Superscalar Architecture 
Three items are always passed into the pipeline. They are the 
instruction code, the predicted target address (including the 
prediaed direction), and the position of the instruction in the 
TB. During trace collection the I-cache supplies the former 
two for eadi instruction, with the third information tagged 
by the TB. During trace emission the TB supplies all three 
items. Feedback from the decoders are sent to the I-cache as 
well as to the TB to update branch predicdon data. 



the TB pointers in the pipeline. These are used for detecting 
misprediaed branches and for managing the TB. Their use 
will become clear later. 

At any instant the processor can be either in the trace col- 
lection mode or in the trace emission mode. During trace 
collection instructions are fetched from the instruction cache 
and the TB simply records the instructions in the instruction 
buffer. Each instruction deliverd by the instruction cache is 
tagged with its position in the TB. When the instructions are 
decoded the entry cache attempts to look up the first back- 
ward branch instruction * in the fetched block. The processor 
must also detect the following three events: 

1. A branch is mispredicted by the instruction cadie: thisis 
done by comparing the resolved target address with the 
predicted target address. When this h^pens the proces- 
sor invalidates the instructions fetched in the IF stage, 
change the PC accordingly, and update the instruction 
buffer content. 

2. A backward branch missed by the entry cache: the ad- 
dress of the backward branch and the next position to 
write in the TB are added to the entry cache. Using this 
infomfiation we can detect a second visit to cache loop 
later. 



backward branch is defined in a limited context They are 
meant to be conditiona] branch instructions using PC-idative ad- 
dress mode. The offset, the, must be negative. 




Figure 9: State TYansition Diagram of the TB co: teller 



3. A backward branch is correctly predicted by the entry 
cache: we say the program now enters the trace^ and 
subsequent instnictions can be delivered from the TB. 
Now the processor switches to the trace emission mode. 

During trace emission instnictions are supplied from the TB. 
Perfect efi&ciency (that is, no branch penalty) can be sustained 
when the processor is executing in this mode. The TB also 
tags the instructions with the prediction information before 
sending them to the instruction registers. Now the processor 
has to watch for the following two events: 

1. A branch is mispredicted by the TB: the program is 
said to have left the trace. Similarly we discard invalid 
instroctionsin the pipeline and update the content of the 
instruction buffer. The processor now switches to the 
trace collection mode. IVace collection by the TB starts 
from where the program left the trace. 

2. A backward branch is conectly predicted by the entry 
cache: the program enters a nested loop. The processor 
updates the content of the TB and enters the iimer loop. 

The content of the TB must be reorganized to avoid bank 
conflicts during the transition from the collection mode to 
the emission mode. The process is e3q)lained in the previous 
section. The state transitions are illustrated in Figure 9. The 
state operations are also sununaxized in T^le 2. 

Performance Evaluation 

All results in this section are expressed in terms of instruction 
fetch ef&ciency. The instruction fetch efficiency is defined 
as the ratio of executed instructions and the instructions de- 
livered to the instruction decode stage in the pipeline. Ihe 
ratio is 1 for prefect ef&ciency where no instructions are 
discareded. 

Three superscalar processor configurations are simulated. 
The first configuration (referred to as CO) uses a standard 
instruction cache and employs no branch prediction scheme. 
The second configuration (referred to as CB) augments the 
first with an branch target buffer (BTB). The BTB takes the 
address of a branch instruction and returns a guess of the 
target address. The third configuration (referred to as CTxx, 
where XX is the ntmiber of entries in the instruction buffer) 
adds a TB on top of the I-cache and the BTB. 
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CORRECT instruction buffer if flag set 
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else 
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ADD its address and TB position to the entry cache 
next state is collection phase 1 

if (first backward branch is taken and in entry cache) 
next state is TB reorg, phase 1 

else if (no mispredicted branch) 

else 

next state is collectioa phase 1 


TB reorg. 
phase 1 


Sb'i'FlKS 

READ first block of instructions from TB 
send them to the instruction registers 
next state is TB reorg. phase 2 


TB reoiiB. 
phase 2 


WRITE them after where the misprediction occurs 

ADVRPTR 

ADVWFIR 

next state is emission phase 1 



Tible 2: State operations of a TB 
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For convenience of discussion we classify the brandi penalty 
into three types. The misprediction penalty is incuned when 
the instruction block fetdied is not the conect one. The 
branch alignment penalty is incurred when a taken branch 
is not the last in the instruction block. Finally the target 
alignment penalty occurs when the branch target is not the 
first in the instruction block. Therefore we expect CB to 
eliminate portions of of misprediction penalty, and CT to 
reduce the branch and target alignment penalty. 

Results 

To avoid explosion in the parameter space we fix the size 
of the BTB entries (128 entries using one prediction bit) 
and the entry cache (8 entries). The choice of BTB size 
is based on the result of the [4] study. A 128-entry BTB 
using one prediction bit yields a prediction rate of 82%. The 
prediction rale can be increase to 88% by using 16 times as 
many entries, but that is immaterial to our purpose here. The 
size of the entry cache is also not important as we observe no 
significant replacement activities in our simulation - space 
in the entry cache is usually available because entries are 
fi^quently invalidated when new data are wrinen into the 
instruction buffer. For the CT configuration we simulate 3 
different instruction buffer sizes (32,64.128). The results are 
given in Table 3 and Table 4. 
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CO 


CB 


CT32 
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CT128 


dhrystone 
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1% 


1% 


1% 



Alignment penalty 



Table 4: Comparision of branch penalty. The results are in 
terms of the penalty per executed instruction. 



indentical results as a 128-entry TB in almost all cases. 

To deal with general purpose programs more effectively we 
suggest the following improvement: 



Program 


CO 


CB 


CT32 


CT64 


CT128 


dhrystone 


41% 


62% 


99% 


99% 


99% 


doduc 


60% 


72% 


78% 


79% 


80% 


dinero 


49% 


68% 


71% 


71% 


71% 


matrix300 


85% 


93% 


99% 


99% 


99% 



Table 3: Comparision of Instruction Fetch Efficiency 



It can be seen that brandi misprediction penalty in CB is 
small (1-9%) when compared with the alignment penalty (6- 
60% without TB). Therefore the net effect of using a BTB is 
to shift the branch bottleneck from the misprediction penalty 
to the alignment penalty. The fetch efi&dency without any 
prediction scheme is simply unacceptable. 

The simulation results also show that instruction trace buffers 
lead to significant improvement for numerical computarion 
programs. The dhrystone and matrixSOO programs fit well 
into this category. For programs with smaU branches and 
loops such as the doduc benchmark, a great portion of the 
alignment penalty can be reduced by a TB. However the 
TB slightly increases the misprediction penalty because it 
anempts to predict several branches as a whole, and extra 
penalt>' is incurred when we enter and leave the trace too 
often. For programs whose behavior dep>ends heavily on the 
inputs the effectiveness of TB is limited. This is indicated in 
the results of the dinero benchmark. In this case the TB is 
activated only 10% of the time. The size of the TB does not 
seem to be an important faaor, l)ecause a 32-entry TB shows 



^this is for corrtciing the branch mispredicted in the previous 
cycle. The CORRECT operation implies a write in Ae instruction 
buffer. 'Hierefore we have to delay it to the next cycle to avoid 
conflicts with the normal WRITE operation 

'this step is skipped if the backward branch is the last instruction 
in the current loop iteration 



♦ Our implementation of TB tends to optimize only the 
innermost loops. This phenomenon can be observed 
when the program leaves the cached trace. At this point 
no anempt is made to recapture the instruction trace of 
an outer loop. We believe this problem can be dealt with 
with a more sophisticated TB control algorithm, albeit 
at the cost of greater complexity. 

• Tht TB mechanism presented in this paper is a pure 
hardware solution and it seems wasteftil not to utilize 
the knowledge available at compile time. For example, 
two iterations are required to identify and store a loop in 
the TB - the first iteration only serves to locate the back- 
ward branch. If the compiler can tag the loop entries 
(possibly by inserting a dununy instructions) with the 
corresponding backward branches, this overhead can be 
saved. A better idea will be to utilize the language se- 
mantics to tell which loops are suitable for optimization. 
For example, if-then statements make good candidates 
for the TB optmization, while switch statements usually 
are not. 



Concluding Remarks 

In this pzper we present the trace buffer technique for opti- 
mizing the instruction fetch performance. Simulation results 
show great improvement for numerical £q>plications, using a 
TB containing only 32 ratries. On the other hand, the gain 
with general purpose applications is limited. To boost TB 
performance further, we suggest that compile-time knowl- 
edge be used to selectively optimize the loops in a program. 

Our discussion so far has been limited to the instruction fetch 
performance. Whether this will lead to overall performance 
improvement is not investagated. However we believe that 
parallelism can be increased when we oveiis^ different in- 
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strucdOD luns. The combined effect of instruction fetch ef- 
ficiency and data dependency seems to be worthy of further 
investigation. 
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